1 00:00:05,120 --> 00:00:03,379 um not sure exactly what I'm gonna talk 2 00:00:06,860 --> 00:00:05,130 about in this talk so I'll give you a 3 00:00:07,970 --> 00:00:06,870 little bit of warning I mean I know what 4 00:00:09,709 --> 00:00:07,980 I'm going to talk about but the 5 00:00:11,330 --> 00:00:09,719 interesting thing is that usually when I 6 00:00:13,280 --> 00:00:11,340 give these kinds of talks it's to a 7 00:00:15,200 --> 00:00:13,290 group who has no idea what an RG is or 8 00:00:18,349 --> 00:00:15,210 why it matters and that's clearly not 9 00:00:20,540 --> 00:00:18,359 this group so I'm going to provide a 10 00:00:23,179 --> 00:00:20,550 brief introduction about myself and 11 00:00:25,429 --> 00:00:23,189 Siler on why it was founded and then I'm 12 00:00:27,170 --> 00:00:25,439 going to talk pretty much about the 13 00:00:29,450 --> 00:00:27,180 types of sloppy experiments that Bob 14 00:00:32,179 --> 00:00:29,460 talked about because in a way as I'll 15 00:00:33,770 --> 00:00:32,189 get to the point of sila Ron is to kind 16 00:00:36,590 --> 00:00:33,780 of have a lot of people conducting a lot 17 00:00:38,569 --> 00:00:36,600 of sloppy experiments and we're trying 18 00:00:40,549 --> 00:00:38,579 to see how the effects work outside of 19 00:00:42,380 --> 00:00:40,559 the lab environment and I think that we 20 00:00:44,450 --> 00:00:42,390 find a lot of correlations outside of 21 00:00:46,250 --> 00:00:44,460 the lab environment with structural data 22 00:00:47,900 --> 00:00:46,260 that's seen in the lab environment so 23 00:00:50,600 --> 00:00:47,910 I'll kind of beat around that a little 24 00:00:51,619 --> 00:00:50,610 bit and show some examples and for 25 00:00:53,930 --> 00:00:51,629 anyone who's interested in conducting 26 00:00:54,979 --> 00:00:53,940 our eg reason experiments I think some 27 00:00:56,720 --> 00:00:54,989 of this information might be a little 28 00:00:57,670 --> 00:00:56,730 bit useful but you can be the judge of 29 00:01:00,709 --> 00:00:57,680 that 30 00:01:03,080 --> 00:01:00,719 so a quick introduction about myself I 31 00:01:04,490 --> 00:01:03,090 started working at the pair lab before I 32 00:01:06,140 --> 00:01:04,500 went to college I was a high school 33 00:01:09,350 --> 00:01:06,150 student and through a long series of 34 00:01:11,270 --> 00:01:09,360 events I took off I decided not to go to 35 00:01:13,219 --> 00:01:11,280 college for a year and I worked at the 36 00:01:16,249 --> 00:01:13,229 pear for a pear lab full-time for one 37 00:01:17,359 --> 00:01:16,259 academic year and two summers and it's 38 00:01:20,030 --> 00:01:17,369 hard for me to think back and remember 39 00:01:22,039 --> 00:01:20,040 this time of my life but basically now 40 00:01:24,109 --> 00:01:22,049 that I'm running cylon I'm constantly 41 00:01:26,330 --> 00:01:24,119 answering questions from people along 42 00:01:28,130 --> 00:01:26,340 the lines of well what happens if I run 43 00:01:29,719 --> 00:01:28,140 the RTG faster or what happens if I have 44 00:01:32,569 --> 00:01:29,729 20 re G's and you know there's a whole 45 00:01:34,789 --> 00:01:32,579 list of these questions and when I 46 00:01:36,710 --> 00:01:34,799 started at the pear lab I kind of view 47 00:01:39,469 --> 00:01:36,720 the effect in that way I thought of it 48 00:01:41,210 --> 00:01:39,479 as I'll refer to it you know it's just 49 00:01:42,380 --> 00:01:41,220 as a causal mechanistic phenomenon the 50 00:01:44,569 --> 00:01:42,390 idea that there's some underlying 51 00:01:46,670 --> 00:01:44,579 mechanism maybe it's physical maybe it's 52 00:01:48,530 --> 00:01:46,680 not but there's a mechanism that can be 53 00:01:50,179 --> 00:01:48,540 thought of as operating on a physical 54 00:01:52,969 --> 00:01:50,189 process I don't really believe that 55 00:01:54,499 --> 00:01:52,979 anymore and a big part of that is 56 00:01:56,300 --> 00:01:54,509 because of my experience at the pear lab 57 00:01:58,609 --> 00:01:56,310 and what I found in the five or six 58 00:02:03,469 --> 00:01:58,619 years following that so I'll talk about 59 00:02:05,480 --> 00:02:03,479 that a little bit but basically what I 60 00:02:07,100 --> 00:02:05,490 what I found was that we were conducting 61 00:02:08,719 --> 00:02:07,110 all these experiments at the lab that 62 00:02:10,609 --> 00:02:08,729 led to interesting bottom line results 63 00:02:12,110 --> 00:02:10,619 but often times they involved hundreds 64 00:02:13,110 --> 00:02:12,120 of thousands of trials or millions of 65 00:02:15,600 --> 00:02:13,120 trials and lots of 66 00:02:17,220 --> 00:02:15,610 for an operator's and one of the things 67 00:02:19,170 --> 00:02:17,230 I did it while I was at the para lab was 68 00:02:20,759 --> 00:02:19,180 I would come in on weekends or spend 69 00:02:22,649 --> 00:02:20,769 some time by myself and just run my own 70 00:02:24,059 --> 00:02:22,659 small experiments I would say you know 71 00:02:25,710 --> 00:02:24,069 what happens if I come in on Saturday 72 00:02:28,979 --> 00:02:25,720 and I'm just gonna do a few trials and 73 00:02:31,199 --> 00:02:28,989 I'm gonna test just arbitrary things 74 00:02:32,940 --> 00:02:31,209 what is the effect of my state of mind 75 00:02:34,770 --> 00:02:32,950 you know it's all very subjective these 76 00:02:36,780 --> 00:02:34,780 are very sloppy experiments in a way I 77 00:02:40,199 --> 00:02:36,790 did do some not sloppy experiments that 78 00:02:43,770 --> 00:02:40,209 bob has helped put together into some 79 00:02:45,479 --> 00:02:43,780 written form and basically in the sloppy 80 00:02:47,490 --> 00:02:45,489 experiments what I was finding was that 81 00:02:49,319 --> 00:02:47,500 you know what over very small numbers of 82 00:02:51,569 --> 00:02:49,329 trials I'm getting terminal levels of 83 00:02:53,300 --> 00:02:51,579 significance that are equivalent to what 84 00:02:56,369 --> 00:02:53,310 we find in these big experiments I'm 85 00:02:58,229 --> 00:02:56,379 feeling subjective indications of my 86 00:02:59,520 --> 00:02:58,239 state of mind or my way of being 87 00:03:01,770 --> 00:02:59,530 actually influencing what's happening 88 00:03:03,330 --> 00:03:01,780 with the device and for people who come 89 00:03:05,460 --> 00:03:03,340 at it with the kinds of assumptions or 90 00:03:07,140 --> 00:03:05,470 ideas that I said I began with which are 91 00:03:09,420 --> 00:03:07,150 that the RDG phenomena is this sort of 92 00:03:11,099 --> 00:03:09,430 causal mechanistic you know there's a 93 00:03:12,809 --> 00:03:11,109 light beam coming out of my mind hitting 94 00:03:15,360 --> 00:03:12,819 the re G and you know it shifts three 95 00:03:18,240 --> 00:03:15,370 bits per 10,000 it would really seem to 96 00:03:19,830 --> 00:03:18,250 be contradictory to say that well we 97 00:03:23,699 --> 00:03:19,840 generated an experiment with six million 98 00:03:25,170 --> 00:03:23,709 trials and you know we've got a z-score 99 00:03:26,580 --> 00:03:25,180 of three well it would seem like a 100 00:03:28,440 --> 00:03:26,590 contradiction to say that this guy can 101 00:03:30,210 --> 00:03:28,450 walk into the room sit down for a couple 102 00:03:31,319 --> 00:03:30,220 of hours conducted a small meaningful 103 00:03:32,849 --> 00:03:31,329 experiment and come out with his e 104 00:03:35,759 --> 00:03:32,859 square of three it almost doesn't make 105 00:03:37,470 --> 00:03:35,769 any sense so anyway I was feeling like I 106 00:03:39,539 --> 00:03:37,480 didn't know if I wanted to go after this 107 00:03:41,610 --> 00:03:39,549 kind of research and a from an basically 108 00:03:43,589 --> 00:03:41,620 an academic perspective because it seems 109 00:03:44,849 --> 00:03:43,599 like our paradigm for evaluating these 110 00:03:47,670 --> 00:03:44,859 kinds of effects was just way too 111 00:03:49,650 --> 00:03:47,680 constrained but at the same time you 112 00:03:51,479 --> 00:03:49,660 know it's having these experiences and 113 00:03:54,000 --> 00:03:51,489 being at the pair lab and kind of seeing 114 00:03:55,920 --> 00:03:54,010 how really effects were that made me 115 00:03:57,270 --> 00:03:55,930 feel like it was just really frustrating 116 00:03:59,280 --> 00:03:57,280 because it's like well I know there's 117 00:04:00,839 --> 00:03:59,290 something that's real at least you know 118 00:04:02,759 --> 00:04:00,849 that's what I believe for my experiments 119 00:04:04,439 --> 00:04:02,769 and it's going to be really hard to 120 00:04:06,960 --> 00:04:04,449 convince society of that in the regular 121 00:04:09,000 --> 00:04:06,970 academic way so I wanted to do something 122 00:04:11,369 --> 00:04:09,010 with the field but you know I didn't 123 00:04:17,060 --> 00:04:11,379 want to spend 40 more years conducting 124 00:04:19,199 --> 00:04:17,070 long experiments so what I did was 125 00:04:22,110 --> 00:04:19,209 started this thing called sila Ron and 126 00:04:23,760 --> 00:04:22,120 yet the basic goal of Seiler on which I 127 00:04:26,370 --> 00:04:23,770 co-founded with the guy named herb Mertz 128 00:04:26,700 --> 00:04:26,380 who was around at the pear lab back in 129 00:04:31,650 --> 00:04:26,710 the seven 130 00:04:32,999 --> 00:04:31,660 these before I was born so we got 131 00:04:34,379 --> 00:04:33,009 together and we started this thing 132 00:04:36,300 --> 00:04:34,389 called Sol around realistically speaking 133 00:04:37,800 --> 00:04:36,310 we did not have much of a goal and there 134 00:04:40,020 --> 00:04:37,810 wasn't that a great business model or 135 00:04:42,330 --> 00:04:40,030 you know this was driven by an interest 136 00:04:43,890 --> 00:04:42,340 in the field and having seen a lot of 137 00:04:45,360 --> 00:04:43,900 things in the research myself so the 138 00:04:46,740 --> 00:04:45,370 basic goal was let's make it possible 139 00:04:48,540 --> 00:04:46,750 for other people can to conduct their 140 00:04:50,820 --> 00:04:48,550 own experiments I've already explained 141 00:04:52,499 --> 00:04:50,830 this but my feeling was that first 142 00:04:54,960 --> 00:04:52,509 person well this is also an important 143 00:04:56,640 --> 00:04:54,970 side point I guess anybody can take the 144 00:04:58,830 --> 00:04:56,650 research and read it and Bob and Brenda 145 00:04:59,879 --> 00:04:58,840 and Roger and all the York and all these 146 00:05:01,920 --> 00:04:59,889 people have worked at the para lab have 147 00:05:03,810 --> 00:05:01,930 said amazing things but if all you do is 148 00:05:05,700 --> 00:05:03,820 you read that you come to know it at a 149 00:05:07,350 --> 00:05:05,710 logical level and you can say well over 150 00:05:08,909 --> 00:05:07,360 these two million trials you know there 151 00:05:12,560 --> 00:05:08,919 was a significant effect but that 152 00:05:15,450 --> 00:05:12,570 doesn't tell anything about you know the 153 00:05:17,100 --> 00:05:15,460 how the effective evolved or how it got 154 00:05:19,050 --> 00:05:17,110 there or what was the actual experience 155 00:05:21,180 --> 00:05:19,060 that led to that effect and what I'm 156 00:05:22,920 --> 00:05:21,190 going to claim is that the experience of 157 00:05:24,659 --> 00:05:22,930 getting there or the subjective process 158 00:05:25,770 --> 00:05:24,669 involved and everybody at pear and other 159 00:05:27,180 --> 00:05:25,780 research in this field have talked about 160 00:05:28,680 --> 00:05:27,190 this already I just want to make it a 161 00:05:31,140 --> 00:05:28,690 little bit more explicit the subjective 162 00:05:33,029 --> 00:05:31,150 experience involved really may play a 163 00:05:35,430 --> 00:05:33,039 fundamental role in driving what kinds 164 00:05:36,899 --> 00:05:35,440 of effects you get so I felt there was a 165 00:05:38,520 --> 00:05:36,909 new need and we needed a new way to 166 00:05:39,930 --> 00:05:38,530 conduct experiments small experiments 167 00:05:41,670 --> 00:05:39,940 that are meaningful we need to explore a 168 00:05:42,899 --> 00:05:41,680 little bit more so the idea with siloam 169 00:05:44,159 --> 00:05:42,909 is we're gonna make re G's and we're 170 00:05:46,439 --> 00:05:44,169 gonna get them out to people and see 171 00:05:47,700 --> 00:05:46,449 what happens also I mean obviously 172 00:05:49,830 --> 00:05:47,710 running an experimental program like 173 00:05:52,680 --> 00:05:49,840 pair is expensive you have to have re G 174 00:05:54,180 --> 00:05:52,690 staff software analysis etc well because 175 00:05:56,010 --> 00:05:54,190 when I was at the para lab another big 176 00:05:57,659 --> 00:05:56,020 part of what I did was make Hardware 177 00:05:58,890 --> 00:05:57,669 write software or conduct analysis I 178 00:06:00,930 --> 00:05:58,900 kind of covered a lot of the spectrum 179 00:06:02,999 --> 00:06:00,940 said well maybe we can kind of box what 180 00:06:04,589 --> 00:06:03,009 the lab did and make it easy for someone 181 00:06:05,610 --> 00:06:04,599 to just by sort of a little prepackaged 182 00:06:07,140 --> 00:06:05,620 thing that they could hook up to their 183 00:06:08,820 --> 00:06:07,150 computer so they can explore this 184 00:06:11,879 --> 00:06:08,830 themselves that's pretty much what we 185 00:06:13,469 --> 00:06:11,889 did so we that was our first product we 186 00:06:15,659 --> 00:06:13,479 call it the reg1 we sell a random event 187 00:06:16,379 --> 00:06:15,669 generator software our analysis tools so 188 00:06:18,719 --> 00:06:16,389 that you can conduct your own 189 00:06:20,279 --> 00:06:18,729 experiments we also nowadays that there 190 00:06:22,110 --> 00:06:20,289 are more people working with us and we 191 00:06:24,800 --> 00:06:22,120 will help people to conduct experiments 192 00:06:28,350 --> 00:06:24,810 design protocols analyze it etc 193 00:06:29,999 --> 00:06:28,360 basically we also were technically a 194 00:06:32,550 --> 00:06:30,009 for-profit company so but it's kind of 195 00:06:34,200 --> 00:06:32,560 weird I mean this was started out of an 196 00:06:35,640 --> 00:06:34,210 interest in the research and we 197 00:06:37,079 --> 00:06:35,650 continued to pretty much conduct 198 00:06:37,950 --> 00:06:37,089 informal research experiments all the 199 00:06:39,860 --> 00:06:37,960 time 200 00:06:41,330 --> 00:06:39,870 the thing here though is that 201 00:06:42,440 --> 00:06:41,340 our experimental paradigm is a little 202 00:06:44,330 --> 00:06:42,450 bit different than let's say the pair 203 00:06:46,010 --> 00:06:44,340 paradigm because our goal is to create 204 00:06:47,120 --> 00:06:46,020 applications we want to just find out 205 00:06:49,850 --> 00:06:47,130 how do these things work in the real 206 00:06:52,220 --> 00:06:49,860 world what can they do what can't they 207 00:06:53,900 --> 00:06:52,230 do I don't really go out of my way to 208 00:06:56,030 --> 00:06:53,910 try to convince anyone that it's real I 209 00:06:57,980 --> 00:06:56,040 think that's extremely important we need 210 00:06:59,690 --> 00:06:57,990 to get the mainstream scientific 211 00:07:03,860 --> 00:06:59,700 paradigm onboard in order to do this 212 00:07:04,850 --> 00:07:03,870 well eventually at least but where we 213 00:07:05,900 --> 00:07:04,860 look at our data a little bit 214 00:07:07,790 --> 00:07:05,910 differently so that's where the 215 00:07:09,020 --> 00:07:07,800 sloppiness part comes into I mean I'm 216 00:07:11,810 --> 00:07:09,030 gonna start bringing some stuff up here 217 00:07:13,910 --> 00:07:11,820 which I could shoot down myself fairly 218 00:07:15,680 --> 00:07:13,920 easily but that's not the point the 219 00:07:17,870 --> 00:07:15,690 point is you'll start to see that there 220 00:07:20,090 --> 00:07:17,880 are sort of anecdotal tendencies that 221 00:07:21,920 --> 00:07:20,100 emerge in the data that are very very 222 00:07:23,000 --> 00:07:21,930 close if not the exact same thing to 223 00:07:26,050 --> 00:07:23,010 what we find in the controlled 224 00:07:28,670 --> 00:07:26,060 laboratory studies so I get to that also 225 00:07:29,870 --> 00:07:28,680 we say we throw some basic assumptions 226 00:07:33,500 --> 00:07:29,880 about science that the wind out the 227 00:07:35,510 --> 00:07:33,510 window that does relate to basically a 228 00:07:36,980 --> 00:07:35,520 little bit of the experimental method we 229 00:07:39,110 --> 00:07:36,990 try to structure the experiments around 230 00:07:40,340 --> 00:07:39,120 the operators rather than try to make 231 00:07:42,020 --> 00:07:40,350 the operators do things around the 232 00:07:43,310 --> 00:07:42,030 experimental protocol I mean it's a 233 00:07:45,620 --> 00:07:43,320 thousand times easier to conduct 234 00:07:46,640 --> 00:07:45,630 experiments with fixed trial lengths you 235 00:07:48,230 --> 00:07:46,650 know everybody operators gonna do 236 00:07:50,780 --> 00:07:48,240 exactly this on this day and on and on 237 00:07:52,220 --> 00:07:50,790 but as an operator myself I feel like 238 00:07:54,590 --> 00:07:52,230 that's not the best way to get an effect 239 00:07:56,330 --> 00:07:54,600 so we were trying to do the best that we 240 00:07:58,670 --> 00:07:56,340 can to develop new ways to conduct 241 00:08:01,430 --> 00:07:58,680 experiments that are both rigorous but 242 00:08:02,990 --> 00:08:01,440 cater to the operators so now I'm gonna 243 00:08:05,120 --> 00:08:03,000 start blowing through this because I may 244 00:08:07,400 --> 00:08:05,130 not be doing so well on time we're 245 00:08:08,960 --> 00:08:07,410 coming out with other things little 246 00:08:11,210 --> 00:08:08,970 robots for example if you want to do 247 00:08:12,740 --> 00:08:11,220 your own robot experiment give us a send 248 00:08:14,060 --> 00:08:12,750 us an email on the next month or two and 249 00:08:16,160 --> 00:08:14,070 we might have something for you and 250 00:08:17,570 --> 00:08:16,170 we're also trying to help people learn 251 00:08:19,760 --> 00:08:17,580 how to better conduct these experiments 252 00:08:21,260 --> 00:08:19,770 what the statistics are what they can 253 00:08:23,900 --> 00:08:21,270 expect et cetera so we're creating some 254 00:08:24,830 --> 00:08:23,910 informational materials and a third part 255 00:08:26,900 --> 00:08:24,840 of what we're doing right now that's 256 00:08:28,910 --> 00:08:26,910 kind of important at least for as far as 257 00:08:30,800 --> 00:08:28,920 researchers are concerned is that we've 258 00:08:32,540 --> 00:08:30,810 always had sort of an online data 259 00:08:34,130 --> 00:08:32,550 collection effort anyone who is 260 00:08:36,170 --> 00:08:34,140 conducting experiments with our re G's 261 00:08:38,720 --> 00:08:36,180 can decide that they want to upload 262 00:08:40,820 --> 00:08:38,730 their data to us and this allows for 263 00:08:44,480 --> 00:08:40,830 some interesting things because we can 264 00:08:46,370 --> 00:08:44,490 start to see which sorts of effects well 265 00:08:49,070 --> 00:08:46,380 in principle we can start to see what 266 00:08:50,510 --> 00:08:49,080 sorts of effects are consistent let's 267 00:08:53,630 --> 00:08:50,520 say across laboratories across 268 00:08:54,140 --> 00:08:53,640 experimenters and which ones are so 269 00:08:55,190 --> 00:08:54,150 maybe 270 00:08:57,770 --> 00:08:55,200 to a particular lab in particular 271 00:08:59,870 --> 00:08:57,780 experimenter in the pair data you know 272 00:09:01,010 --> 00:08:59,880 there's some evidence that and I'll talk 273 00:09:02,330 --> 00:09:01,020 about this a little bit later but 274 00:09:04,190 --> 00:09:02,340 there's evidence that during time 275 00:09:05,060 --> 00:09:04,200 periods of enthusiasm excitement etc 276 00:09:06,650 --> 00:09:05,070 there might have been slightly better 277 00:09:08,390 --> 00:09:06,660 effects and then you tell the operators 278 00:09:10,190 --> 00:09:08,400 okay it's time to replicate now we don't 279 00:09:11,750 --> 00:09:10,200 believe you effectively or something 280 00:09:13,940 --> 00:09:11,760 like that and the results aren't as good 281 00:09:16,490 --> 00:09:13,950 and that comes up all the time at Siler 282 00:09:17,780 --> 00:09:16,500 on so get into that I'm alright this 283 00:09:19,820 --> 00:09:17,790 claimer I'm just gonna fly through this 284 00:09:21,740 --> 00:09:19,830 I already said I'm talking about sloppy 285 00:09:22,820 --> 00:09:21,750 stuff so not everything I'm going to say 286 00:09:24,710 --> 00:09:22,830 is gonna come from well-controlled 287 00:09:26,510 --> 00:09:24,720 studies but I hope we can get around 288 00:09:28,160 --> 00:09:26,520 that temporarily and also I don't feel 289 00:09:30,290 --> 00:09:28,170 bad about this because kind of as Bob 290 00:09:31,970 --> 00:09:30,300 said there is a lot of precedent for 291 00:09:33,980 --> 00:09:31,980 being able to conduct good experiments 292 00:09:36,890 --> 00:09:33,990 over short time spans with under the 293 00:09:38,570 --> 00:09:36,900 least ideal circumstances all right so 294 00:09:40,310 --> 00:09:38,580 I'm gonna rehash some pair ideas just to 295 00:09:41,570 --> 00:09:40,320 give everybody some background facts do 296 00:09:43,490 --> 00:09:41,580 not appear to be dependent on the 297 00:09:46,220 --> 00:09:43,500 underlying physical phenomena basically 298 00:09:49,310 --> 00:09:46,230 speaking and again you can't say this 299 00:09:50,420 --> 00:09:49,320 for sure basically speaking there's at 300 00:09:52,730 --> 00:09:50,430 pair lab you have their end of 301 00:09:54,440 --> 00:09:52,740 mechanical cascade the re G's the robots 302 00:09:55,610 --> 00:09:54,450 are different types of re G's they all 303 00:09:57,770 --> 00:09:55,620 seem to at the end of the day produce 304 00:09:59,510 --> 00:09:57,780 results with similar structure Seiler on 305 00:10:01,220 --> 00:09:59,520 we've tried different types of re G's 306 00:10:02,750 --> 00:10:01,230 since we are an RTG company we've just 307 00:10:04,220 --> 00:10:02,760 used different physical processing 308 00:10:05,990 --> 00:10:04,230 methods the end of the day we see 309 00:10:07,400 --> 00:10:06,000 similar things other background finding 310 00:10:08,660 --> 00:10:07,410 Bob talked about the robot experiment 311 00:10:11,330 --> 00:10:08,670 operators can create effects on a 312 00:10:13,880 --> 00:10:11,340 short-term basis student projects tend 313 00:10:16,070 --> 00:10:13,890 to get good experiments etcetera I'm 314 00:10:18,710 --> 00:10:16,080 gonna skip that so some quick points 315 00:10:20,000 --> 00:10:18,720 database sighs I'm going to skip this as 316 00:10:21,140 --> 00:10:20,010 well because I guess I already talked 317 00:10:22,610 --> 00:10:21,150 about it a little bit but I would 318 00:10:24,530 --> 00:10:22,620 suggest that if anyone is going to 319 00:10:26,600 --> 00:10:24,540 conduct experiments that you conduct 320 00:10:29,090 --> 00:10:26,610 short well controlled experiments with a 321 00:10:30,380 --> 00:10:29,100 very clear well-defined purpose meaning 322 00:10:32,390 --> 00:10:30,390 you don't need to sit around and 323 00:10:34,700 --> 00:10:32,400 generate you know two million trials in 324 00:10:36,530 --> 00:10:34,710 order to come up with a conclusion 325 00:10:37,520 --> 00:10:36,540 saying that there was an effect in fact 326 00:10:40,250 --> 00:10:37,530 I think that may be counterproductive 327 00:10:42,200 --> 00:10:40,260 and I'll get to that in a second serious 328 00:10:43,820 --> 00:10:42,210 position effects Bob John talked about 329 00:10:45,440 --> 00:10:43,830 this in the robot paper you saw the 330 00:10:47,480 --> 00:10:45,450 first series of the second operator 331 00:10:48,980 --> 00:10:47,490 being very significant you saw a second 332 00:10:52,660 --> 00:10:48,990 series that was inverted with a 333 00:10:54,710 --> 00:10:52,670 comparable effect size um we find this 334 00:10:56,210 --> 00:10:54,720 sort of in the real world occurring at 335 00:10:58,670 --> 00:10:56,220 many many levels within the experiment 336 00:11:00,290 --> 00:10:58,680 someone they do their first series of 337 00:11:02,420 --> 00:11:00,300 the day maybe the second one will be 338 00:11:03,680 --> 00:11:02,430 inverted they do their second day of an 339 00:11:07,280 --> 00:11:03,690 experiment and there may be a tendency 340 00:11:07,910 --> 00:11:07,290 towards a version inversion so that's 341 00:11:10,069 --> 00:11:07,920 pretty 342 00:11:11,840 --> 00:11:10,079 their philosophical confounds related to 343 00:11:13,550 --> 00:11:11,850 this or the philosophical issues to be 344 00:11:15,350 --> 00:11:13,560 aware of a lot of times when you tell an 345 00:11:16,490 --> 00:11:15,360 operator in my experience when an 346 00:11:18,530 --> 00:11:16,500 operator finds out that they've been 347 00:11:20,690 --> 00:11:18,540 doing really well we'll see an inversion 348 00:11:22,340 --> 00:11:20,700 shortly after this occurs that almost 349 00:11:23,389 --> 00:11:22,350 all experimental levels so I want to 350 00:11:24,800 --> 00:11:23,399 mention that and also that it's 351 00:11:27,710 --> 00:11:24,810 particularly relevant to same lab 352 00:11:28,430 --> 00:11:27,720 replications you can't well I'll get to 353 00:11:30,620 --> 00:11:28,440 that later 354 00:11:32,449 --> 00:11:30,630 bottom line significance do not expect 355 00:11:34,100 --> 00:11:32,459 to find every creasing increasing levels 356 00:11:36,259 --> 00:11:34,110 of statistical significance some 357 00:11:37,460 --> 00:11:36,269 experiments our operator data databases 358 00:11:38,810 --> 00:11:37,470 seem do they get to a level of 359 00:11:39,230 --> 00:11:38,820 significance and an mo you sit there 360 00:11:41,120 --> 00:11:39,240 forever 361 00:11:44,060 --> 00:11:41,130 it's kind of what bob was talking about 362 00:11:46,040 --> 00:11:44,070 with the point O 5 effect and actually 363 00:11:47,630 --> 00:11:46,050 it's Sylar on I started by saying point 364 00:11:50,930 --> 00:11:47,640 O 5 isn't good enough we need to see 365 00:11:53,060 --> 00:11:50,940 bigger effects and actually again very 366 00:11:54,860 --> 00:11:53,070 anecdotal it seems like that sort of 367 00:11:57,290 --> 00:11:54,870 hanging position you know where is the 368 00:11:59,420 --> 00:11:57,300 data going to get stopped is related to 369 00:12:01,220 --> 00:11:59,430 the particular experimenter or the 370 00:12:03,170 --> 00:12:01,230 operator now I'm not saying that you can 371 00:12:04,610 --> 00:12:03,180 say oh my significance criteria is one 372 00:12:07,639 --> 00:12:04,620 in a billion and that's going to all of 373 00:12:08,870 --> 00:12:07,649 a sudden give you great results but 374 00:12:11,420 --> 00:12:08,880 there's some interesting stuff there and 375 00:12:12,800 --> 00:12:11,430 finally purpose it's very important that 376 00:12:14,360 --> 00:12:12,810 you're generating data for a reason we 377 00:12:16,220 --> 00:12:14,370 find that when people are let's say you 378 00:12:17,449 --> 00:12:16,230 running an R eg during a job interview 379 00:12:18,829 --> 00:12:17,459 or something they say hey how is my 380 00:12:20,780 --> 00:12:18,839 residence with the interview I have 381 00:12:23,300 --> 00:12:20,790 people who go dancing and they run it 382 00:12:25,280 --> 00:12:23,310 when they go dancing all these different 383 00:12:26,870 --> 00:12:25,290 situations we tend to see much larger 384 00:12:27,860 --> 00:12:26,880 effects that are much more relevant when 385 00:12:30,470 --> 00:12:27,870 people are doing it for that kind of 386 00:12:32,420 --> 00:12:30,480 purpose then when for example I say 387 00:12:34,610 --> 00:12:32,430 there's a graduate student / slave / 388 00:12:36,050 --> 00:12:34,620 employee problem this is where you see 389 00:12:37,970 --> 00:12:36,060 you've already decided that your 390 00:12:39,620 --> 00:12:37,980 experiment is going to be 500,000 trials 391 00:12:40,970 --> 00:12:39,630 long and you start telling it you start 392 00:12:42,500 --> 00:12:40,980 cracking the whip and said guys we need 393 00:12:44,269 --> 00:12:42,510 more data come on go generate some data 394 00:12:46,730 --> 00:12:44,279 so people go into the room and they 395 00:12:48,889 --> 00:12:46,740 start generating it but you know it just 396 00:12:50,329 --> 00:12:48,899 doesn't do anything so I need a quick 397 00:12:51,350 --> 00:12:50,339 example and I'm gonna close with this 398 00:12:55,819 --> 00:12:51,360 example because I only have about a 399 00:12:58,040 --> 00:12:55,829 minute left hum so long story short we 400 00:12:59,389 --> 00:12:58,050 had an operator who had generated two 401 00:13:01,189 --> 00:12:59,399 million seven hundred fifty two thousand 402 00:13:03,019 --> 00:13:01,199 single bit trials the point is their 403 00:13:04,550 --> 00:13:03,029 result is their overall database over 404 00:13:05,990 --> 00:13:04,560 the course of many months we just took a 405 00:13:07,880 --> 00:13:06,000 snapshot at the time we were doing this 406 00:13:09,199 --> 00:13:07,890 it was statistically significant you 407 00:13:10,730 --> 00:13:09,209 know like the z-score of about a two 408 00:13:13,790 --> 00:13:10,740 point five eight I think I've yeah 409 00:13:15,079 --> 00:13:13,800 two point five eight was marginally it 410 00:13:17,180 --> 00:13:15,089 was statistically significant but it's 411 00:13:19,010 --> 00:13:17,190 two million some odd trials so this 412 00:13:20,600 --> 00:13:19,020 person spent 96 hours or something 413 00:13:21,230 --> 00:13:20,610 generating this and I'm having an 414 00:13:23,210 --> 00:13:21,240 argument with the 415 00:13:25,040 --> 00:13:23,220 operator kind of saying he was saying 416 00:13:26,449 --> 00:13:25,050 well I can train I can get better we can 417 00:13:28,850 --> 00:13:26,459 get better results and I was saying I 418 00:13:30,620 --> 00:13:28,860 believe that you can train etc but I 419 00:13:32,690 --> 00:13:30,630 think that we're gonna get hit by this 420 00:13:34,490 --> 00:13:32,700 sort of arbitrary significance level 421 00:13:37,730 --> 00:13:34,500 issue where you know the significance 422 00:13:39,860 --> 00:13:37,740 level is 0.05 so you get stuck at 0.05 423 00:13:42,139 --> 00:13:39,870 and I said for you you're gonna get 424 00:13:43,850 --> 00:13:42,149 stuck at 2 point something you know this 425 00:13:45,530 --> 00:13:43,860 a lot most of it I said most of the data 426 00:13:47,720 --> 00:13:45,540 that you've showed me anecdotally in 427 00:13:50,269 --> 00:13:47,730 real life has been you know disease of 428 00:13:51,769 --> 00:13:50,279 2.1 the 2.5 issue no they're almost 429 00:13:53,840 --> 00:13:51,779 always in the twos on a great day you 430 00:13:55,250 --> 00:13:53,850 might get a three but and it does seem 431 00:13:59,210 --> 00:13:55,260 like the persons doing it pretty much on 432 00:14:01,250 --> 00:13:59,220 demand roughly speaking but whatever so 433 00:14:02,570 --> 00:14:01,260 he sends me an email and it was funny we 434 00:14:06,769 --> 00:14:02,580 were joke and he said you'll pay for 435 00:14:07,910 --> 00:14:06,779 this so he sends me an email that says 436 00:14:10,070 --> 00:14:07,920 you'll pay for this and then about a 437 00:14:11,449 --> 00:14:10,080 half hour later he sends me this this 438 00:14:12,740 --> 00:14:11,459 result from our program it's gonna be 439 00:14:14,420 --> 00:14:12,750 really hard for me to fly through it but 440 00:14:16,280 --> 00:14:14,430 the bottom line point is that in 30 441 00:14:18,350 --> 00:14:16,290 minutes he generated an re G effect with 442 00:14:21,110 --> 00:14:18,360 the result of z-score of 2 over 30 443 00:14:22,730 --> 00:14:21,120 minutes so now he's got a result with 444 00:14:25,660 --> 00:14:22,740 statistical significance that's almost 445 00:14:28,699 --> 00:14:25,670 comparable to his overall database but 446 00:14:30,199 --> 00:14:28,709 he did it in 30 minutes so he said he 447 00:14:31,850 --> 00:14:30,209 said you'll pay he sent me this and he 448 00:14:33,470 --> 00:14:31,860 was like I showed you and I was like 449 00:14:36,560 --> 00:14:33,480 wait a second no you didn't you're just 450 00:14:37,639 --> 00:14:36,570 proving my point you got the same level 451 00:14:40,340 --> 00:14:37,649 of significance that you had in the 452 00:14:42,079 --> 00:14:40,350 overall database etc and so he went away 453 00:14:43,280 --> 00:14:42,089 for a little bit and then he sends me 454 00:14:44,810 --> 00:14:43,290 another email a little while later with 455 00:14:48,769 --> 00:14:44,820 a screenshot attached and it says I hope 456 00:14:51,560 --> 00:14:48,779 you're happy and this this red means 457 00:14:54,710 --> 00:14:51,570 that went backwards to intention and his 458 00:14:56,360 --> 00:14:54,720 overall z-score was negative 2.1 it was 459 00:14:58,280 --> 00:14:56,370 a much larger number of trials the net 460 00:14:59,630 --> 00:14:58,290 effect is that you know this is an 461 00:15:01,970 --> 00:14:59,640 anecdote but the net effect is his 462 00:15:05,329 --> 00:15:01,980 overall database ended up just about 463 00:15:07,670 --> 00:15:05,339 where it started and that's it so I'm 464 00:15:09,949 --> 00:15:07,680 out of time I'm gonna stop here I tried 465 00:15:16,130 --> 00:15:09,959 to pace myself didn't do a great job but 466 00:15:17,810 --> 00:15:16,140 thank you John oh and this point real 467 00:15:19,460 --> 00:15:17,820 quick just so you know the point is that 468 00:15:20,930 --> 00:15:19,470 your entire database could be comprised 469 00:15:22,699 --> 00:15:20,940 of these small meaningful Purpose Driven 470 00:15:24,710 --> 00:15:22,709 effects and that that's going to need to 471 00:15:27,530 --> 00:15:24,720 be pulled out somehow later John this is 472 00:15:29,840 --> 00:15:27,540 a great idea I you should be applauded 473 00:15:32,150 --> 00:15:29,850 for this it's absolutely wonderful I'd 474 00:15:33,519 --> 00:15:32,160 like to see your hardware in every High 475 00:15:35,829 --> 00:15:33,529 School in the country 476 00:15:37,960 --> 00:15:35,839 um what's your price point for your 477 00:15:40,470 --> 00:15:37,970 hardware right now we have what's called 478 00:15:42,939 --> 00:15:40,480 the light Edition and that uses 479 00:15:44,860 --> 00:15:42,949 basically it has less research oriented 480 00:15:47,439 --> 00:15:44,870 analysis capabilities and that's I think 481 00:15:49,480 --> 00:15:47,449 we have it at 245 right now and the 482 00:15:51,610 --> 00:15:49,490 higher end is $5.99 and that has 483 00:15:52,960 --> 00:15:51,620 additional shielding go through some 484 00:15:54,519 --> 00:15:52,970 more calibration and there are more 485 00:15:57,220 --> 00:15:54,529 advanced features that researchers would 486 00:15:58,720 --> 00:15:57,230 want to use we hope I mean this gets 487 00:16:00,879 --> 00:15:58,730 bigger and we could produce more of them 488 00:16:03,220 --> 00:16:00,889 it would drive the cost down but not yet 489 00:16:06,100 --> 00:16:03,230 we're not there yet you know I also want 490 00:16:08,920 --> 00:16:06,110 to compliment you and it's really to see 491 00:16:11,769 --> 00:16:08,930 young people being creative and going 492 00:16:14,170 --> 00:16:11,779 beyond where their their professors and 493 00:16:15,489 --> 00:16:14,180 teachers have taken them I had the 494 00:16:16,869 --> 00:16:15,499 privilege of working with the student at 495 00:16:19,360 --> 00:16:16,879 the University of Arizona his name is 496 00:16:21,309 --> 00:16:19,370 Lonnie Nelson and his PhD and for his 497 00:16:24,489 --> 00:16:21,319 master's degree he did a single-subject 498 00:16:26,230 --> 00:16:24,499 experiment a total of 5 experiments 499 00:16:27,910 --> 00:16:26,240 which he actually had approved through 500 00:16:32,619 --> 00:16:27,920 his best resume it II ultimately 501 00:16:34,420 --> 00:16:32,629 published in JSC on individual his own 502 00:16:36,400 --> 00:16:34,430 consciousness people she tracked on a 503 00:16:37,960 --> 00:16:36,410 moment-to-moment basis and how it 504 00:16:39,910 --> 00:16:37,970 related to the RT and did a whole series 505 00:16:41,710 --> 00:16:39,920 of experiments and what his personal 506 00:16:43,829 --> 00:16:41,720 observation was exactly what you've 507 00:16:46,660 --> 00:16:43,839 reported which is the spontaneous 508 00:16:49,240 --> 00:16:46,670 capability for making discoveries and 509 00:16:50,980 --> 00:16:49,250 also for having enthusiasm and also the 510 00:16:53,410 --> 00:16:50,990 whole issue of humility versus arrogance 511 00:16:54,819 --> 00:16:53,420 and all these factors and the only way 512 00:16:57,069 --> 00:16:54,829 those discoveries are ever going to be 513 00:16:59,139 --> 00:16:57,079 made is if people have access to this 514 00:17:01,449 --> 00:16:59,149 technology in an interesting fashion and 515 00:17:03,549 --> 00:17:01,459 you're making that possible so I just 516 00:17:05,949 --> 00:17:03,559 also want to personally thank you and 517 00:17:07,659 --> 00:17:05,959 hope this gets to till not just lots of 518 00:17:10,990 --> 00:17:07,669 high schools but hopefully to private 519 00:17:12,610 --> 00:17:11,000 homes and and so on thank you um and as 520 00:17:14,230 --> 00:17:12,620 a quick note on that I didn't get to an 521 00:17:15,850 --> 00:17:14,240 in in presentation but we actually have 522 00:17:17,169 --> 00:17:15,860 started coming up with new terms they're 523 00:17:18,880 --> 00:17:17,179 not really new terms but for example my 524 00:17:21,069 --> 00:17:18,890 partner use of the word micro psychology 525 00:17:22,720 --> 00:17:21,079 basically this idea that on a very small 526 00:17:25,210 --> 00:17:22,730 moment-to-moment basis there are 527 00:17:27,010 --> 00:17:25,220 fluctuations within operators let's say 528 00:17:28,630 --> 00:17:27,020 consciousness that manifests itself on 529 00:17:30,639 --> 00:17:28,640 the re G and if we kind of use 530 00:17:32,680 --> 00:17:30,649 experimental methods that are too blunt 531 00:17:34,060 --> 00:17:32,690 we'll never that would never shake out 532 00:17:36,520 --> 00:17:34,070 of the data so we're really working on 533 00:17:39,190 --> 00:17:36,530 figuring that out John I've got one 534 00:17:41,320 --> 00:17:39,200 suggestion for you change your last name 535 00:17:45,380 --> 00:17:41,330 to gates 536 00:17:47,210 --> 00:17:45,390 well we'll see what happens but I'd like 537 00:17:49,370 --> 00:17:47,220 to give you back 45 seconds or something 538 00:17:52,669 --> 00:17:49,380 and lets you and invite you to tell us 539 00:17:55,970 --> 00:17:52,679 all the most exciting you know 540 00:17:58,310 --> 00:17:55,980 sloppy little study you ever did so that 541 00:18:01,850 --> 00:17:58,320 we can get what fun it was for you 542 00:18:03,740 --> 00:18:01,860 that's tough see that's the thing with 543 00:18:05,899 --> 00:18:03,750 the enthusiasm in the experiments it's 544 00:18:07,580 --> 00:18:05,909 almost a contradiction with the idea of 545 00:18:09,649 --> 00:18:07,590 replicability in the first place so what 546 00:18:11,630 --> 00:18:09,659 happens is something will spontaneously 547 00:18:13,370 --> 00:18:11,640 emerge that leads us to conducting 548 00:18:14,480 --> 00:18:13,380 experience or for example I was sort of 549 00:18:16,659 --> 00:18:14,490 having that argument with the operator 550 00:18:18,890 --> 00:18:16,669 what's gone the other way before where 551 00:18:20,299 --> 00:18:18,900 someone has said to me oh well you know 552 00:18:22,250 --> 00:18:20,309 the effect sizes can't be that large 553 00:18:23,659 --> 00:18:22,260 everybody z-scores are such and such so 554 00:18:26,000 --> 00:18:23,669 one time for awhile I conducted 555 00:18:27,980 --> 00:18:26,010 experiments where I would only generate 556 00:18:29,659 --> 00:18:27,990 a bit or two and then I would go away 557 00:18:30,560 --> 00:18:29,669 and I try to completely forget about it 558 00:18:32,390 --> 00:18:30,570 and then I would come back and I 559 00:18:33,649 --> 00:18:32,400 generate another bit or two etc and the 560 00:18:36,860 --> 00:18:33,659 point was that at the end of the day 561 00:18:38,930 --> 00:18:36,870 over maybe it was a hundred or 150 or 562 00:18:40,789 --> 00:18:38,940 something trials he's painstaking wee 563 00:18:41,720 --> 00:18:40,799 difficult to generate bits we had a 564 00:18:43,880 --> 00:18:41,730 result that was extraordinarily 565 00:18:45,260 --> 00:18:43,890 statistically significant but the point 566 00:18:47,360 --> 00:18:45,270 there was I was saying look my hit rates 567 00:18:49,070 --> 00:18:47,370 like 80 percent so and I was trying to 568 00:18:53,659 --> 00:18:49,080 prove the point that doesn't make any 569 00:18:55,010 --> 00:18:53,669 sense to talk about you know to talk 570 00:18:56,990 --> 00:18:55,020 about things in terms of well what is 571 00:18:58,340 --> 00:18:57,000 the bit per bit effect size etc so I 572 00:18:59,990 --> 00:18:58,350 think there are situations where at 573 00:19:00,830 --> 00:19:00,000 least within our own inner group people 574 00:19:02,390 --> 00:19:00,840 kind of challenge one another 575 00:19:04,580 --> 00:19:02,400 scientifically and then we get like 576 00:19:05,810 --> 00:19:04,590 enthusiastic about it and we want to 577 00:19:08,510 --> 00:19:05,820 make a point but it's not that serious 578 00:19:13,580 --> 00:19:08,520 it's all in fun and neat stuff comes out 579 00:19:18,049 --> 00:19:13,590 of it last question I know it's 580 00:19:21,590 --> 00:19:18,059 wonderful beautiful presentation I love 581 00:19:24,760 --> 00:19:21,600 your spontaneity but I'm asking a minor 582 00:19:27,500 --> 00:19:24,770 question did you think of carrying this 583 00:19:30,649 --> 00:19:27,510 experiment in terms of the times of day 584 00:19:34,789 --> 00:19:30,659 is only controlled by certain times of 585 00:19:37,390 --> 00:19:34,799 the arm we do have a 24 hours would you 586 00:19:40,159 --> 00:19:37,400 see some major variations a major 587 00:19:42,230 --> 00:19:40,169 changes I have never seen anything like 588 00:19:43,730 --> 00:19:42,240 that but we do track things like we do 589 00:19:45,260 --> 00:19:43,740 try have time information in all of our 590 00:19:47,149 --> 00:19:45,270 data so we could easily look at it we 591 00:19:48,620 --> 00:19:47,159 try to track almost as much as we can so 592 00:19:51,169 --> 00:19:48,630 that after the fact if we really wanted 593 00:19:53,659 --> 00:19:51,179 to we can go back over it and look for 594 00:19:56,800 --> 00:19:53,669 such things thank you John for a